DOCUMENT RESUME 



ED 228 194 

AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 
REPORT NO 
PUB DATE 
CONTRACT 
- NOTE 

AVAILABLE FROM 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



SP 022 031 



Yinger, Robert J.; Clark, Christopher M. . 
Understanding Teachers* Judgments about Instruction: 
The Task, the Method, and the Meaning. 
Michigan State Univ., East Lansing. Inst, for 
Research on Teaching. 

National Inst, of Education (ED), Washington, DC. 

IRT-RS-121 

Dec 82 

400-81-0014 

40p. 

Institute for Research on Teaching, College of 
Education, Michigan State University, 252 Erickson 
Hall, East Lansing, MI 48824 ($4.00). 
Reports - Researdh/Technical (143) 

MF01/PC02 Plus Postage. 

Cognitive Processes; *Course Evaluation; *Decision 
Making; Difficulty Level; Elementary School Teachers; 
Evaluation Criteria; Evaluation Methods; *Evaluative 
Thinking; Grade 4; Grade* 5; Instructional Materials; 
I 

Act 

♦Teacher Attitudes; 



111 UK 1 My , Ui ouc -x f uiouc «< , » ..^ *« *» -» w — - --— ' 

ntermediate Grades; *Language Arts; *Learning 
activities; Skill Development; Student Participation; 
- — Teaching Methods 



ABSTRACT 

Six experienced teachers of fourth and fifth grades 
read and made judgments about the appropriateness, attractiveness, 
usefulness,' and effectiveness of each of 32 short descriptions of 
language T arts activities. The activity descriptions had been edited 
to vary systematically on five features: amount of student 
involvement; difficulty for students; integration of multiple skills 
or subject matters ; demand on 'the teacher; and fit between stated 
purpose and instructional process. Each teacher thought aloud during 
the judgment process; these deliberations were audiotaped and 
transcribed. A policy-capturing analysis indicated that only small 
proportions of variance in teacher judgments could be explained in 
terms o^the five manipulated cues. A process-tracing analysis of the 
protocols of two of the six teachers revealed that they were 
attending to several activity features, in addition to the )five 
mentioned above. The process-tracing analysis also provided a > 
description of how the teacher^ defined and modified the -judgment 
task and drew upon their own 'professional experiences to comprehend, 
edit, and evaluate the activity descriptions. The process-tracing 
approach appears to be a more promising method of inquiry into 
complex teacher-judgment tasks than is the policy-capturing approach. 
(Authors/JD) 
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Abstract , 

Six experienced teachers of fourth and fifth grades read and 
n^de Judgments about the appropriateness, attractiveness, usefulness, 
and effectiveness of each of 32 short descriptions -of language-arts 
activities. The activity descriptions had been edited to vary sys. 
tematically on five features: amount of student involvement, dif- 
ficulty for students, integration of multiple skills or subject 
otters, demand on the teacher, and fit between stated purpose and 
instructional process. Each teacher thought aloud during the judgment 
process. These deliberations were audiotaped and transcribed. A 
policy-capturing analysis indicated that only small proportions, of 
variance in teacher judgments could be explained in terms of the five 
rnanipulated cues. A process-tracing analysis of the protocols of 
two of the six teachers revealed that they were attending to several 
activity features in addition to the five mentioned above. The " 
process-tracing analysis also provided a description of how the 
teachers defined and modified the judgment task and dtew upon their 
own professional experiences to comprehend, edit, and evaluate the 
activity descriptions.- The authors concluded the process-tracing 
approach is a more promising method of inquiry into realistically 
complex teacher-judgment tasks than is the policy-capturing approach. 
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Robert J. Yinger and' Christopher M. Clark 

Sound judgment is a critical skill in any profession, be it law, 
medicine, engineering, architecture, or teaching. Because of its 
importance^ the judgment process has become the.object of psycho- 
logical research in recent years (Sl'ovic, Fischoff , & Li Atenstein, 
.1977). In the research 6n teacher thinking, teacher judgment has 
emerged as one of several foci of research (Clark & Yinger, 1977; 
Shavelson & Stearn,, 1981) . This paper reports the results of a 
study of teacher judgment during an instructional-activities selec- 
tion process. 

The most frequently used approach to studying and representing 
judgment processes is policy capturing (Slovic & Liechtenstein, 1971; 
' Shulman & Elstein, 1975). . It involves the use of. a simple mathemati- 
cal model (usually linear) and attempts to reproduce the inferential ' 
responses 6 a particular judge. Of central interest in this paradigm 
is how judges weighed combine information from cues or features of 

the objects to be judged. t 

This approach has been used widely in research on teacher thinking. 
Recently, policy-capturing models were used to represent teachers' 

• l^is p Iper was presented at the annual meeting of the American 
Educatfonal P ResLrch Association in New York City, March 1982. 

Robert Yinger is a former IRT research. : with the Teach. : Plan- 
ning Project and an assistant P^ssor of ^ucation at the ^ ^ 
of .Cincinnati. Christopher Cl«k coordinated the IR ^ 
ning Project and is an associate professor or .auc 
State University. 



judgments of the characteristics of effective teachers (Anderson, 
1977), /effective classroom management (Cone, Note 1), classroom 
organization (Borko, Note 2), instructional strategies (Russo, Note 
3), and instructional content (Floden, Porter, ' Schmidt, Freeman, & 

Schwille, 1981). 

Process tracing, another approach to modeling problem solving 
and decision making, has rarely been used to study teacher judgment. 
Methods involving process- tracing approach very differently the in- 
vestigation and. representation of thinking processes. Since intro- 
spective reports of many judges seem to indicate the presence of a 
complex and configural judgment process, process tracing begins with a 
complex .representation of the' judgment using verbal protocols and 
simplifyingthe mental processes involved by .using decision trees .or 
networks, flow diagrams, and so on. 

Process tracing is now commonly used to study problem-solving . 
presses (e.g., Newell & Simon, 1972). It has also proven success- 
ful in modeling chess thinking (de Groot, 1965), medical diagnosis 
(ElStein, Shulman, 6 Sprafka, 1979), business decision making (Clark- 
. son,. 1962), and consumer choice (Bettman, 1971). ,In research on 

teaching, Yinger (1980, Note 4) used process- tracing methods to char- 
acterize teacher planning. 

The study reported here is part of a series of studies investi- 
gating teacher judgment during the selection of instructional mate- 
rials. These studies include one that identifies factors influencing 
the selection of instructional activities (Clark, Yinger, & Wildfong, 
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Note. 5), a policy-capturing study of teacher judgment (Yinger, 
Clark, & Mondol, Note" 6), a process-tracing study of teacher judg- 
ment, a feature analysis of preferred instructional activities i-and 
an analysis of teachers* self- reported judgment processes. This 
paper presents and discusses a representation of the task of teachers 
selecting classroom activities (that representation described by pro- 
cess-tracing methods) and compares thi3 representation with that gen- 
erated by the policy-capturing method. 

The major research questions guiding this study are: 



1. What factors do teachers take into account when 
selecting instructional activities? 

2. What thinking processes are involved in arriving 
at these judgments? 

3. What are the relative effectivenesses of policy- 
capturing and process-tracing methods for modeling 
these types of judgment? 



Method 

Subject 

Six fourth- and fifth- grade teachers from two Michigan school 
districts volunteered to participate in this study. The three male 
"and three female teachers ranged in age from the mid-twenties to^the 
mid-thirties and averaged 4.8 years of teaching experience (the range 
was, from 4 to 6 years) . Four teachers taught in self-contained 
classrooms; two others team taught. The participants were paid for 
theix participation. 



Materials 

Thirty- two one- or two-vage descriptions of language-arts writing 
activities were derived from a catalogue of language-arts activities 
for upper elementary classrooms (Forte, Frank, & McKenzie, 1973). 
The activity descriptions, presented in the same -general format, con- 
sisted of an activity title, a one or two sentence statement of the 
purpose of the activity, anji a listing of the steps involved in 
planning and conducting the activity. 

The edited activity description reflected five dimensions found 
to be important in teachers 1 judgments of the quality of language- arts 
Instructional materials (Clark, Yinger, & Wildfong, Note'5)., These 
dimensions, or cues, were 

1. student involvement, 

2. difficulty for students, / 

3. integration with other skills or subject matter, 
A. demand on the teacljer, *and 

5.' fit between the stated purpose and described instruc- 
tional process. 

• 4 

The 32 activity descriptions represented a full factoral matrix 
of high and lew values for each cue. After editing, four . researchers 
examined each activity description to determine whether the cue levels 
had been set in the intended configuration. Disagreements among the 
researchers on this task were resolved by further editing and nego- 
tiation. 

We hypothesized that the evaluation and' selection of instruc- 
tional materials is not a one- judgment process but rather a series 
of judgments leading to a final decision to implement or not to 
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implement an activity. (As we will show below, this even proved to 

' ' ' 

be an oversimplified conception of the activities involved in the 

judgment task.) To investigate thij? "process and to better represent 

the complexity of the judgment task, each participant was asked to 

£> 

respond- to four questions about each activity* On the back of each 
activity description the questions were ^stated along with a nine-point 
continuum to record each Response. The four questions or judgments 
to be made about each activity were as follows: 

1. "How attractive is this activity to you? 

2. How appropriate is this activity as part of a 

• catalogue of language arts activities for fourth 
and fifth grade teachers? 

3. How likely would you be to use this activity as 
it is JLn. your present classroom? 

4. How effective do you think this activity would be 
f6r your students? 

*\ 

Procedure ? 
% — — — — — 

4 I 

In two judgment sessions, each lasting approximately one hour, 
each teacher first was given information describing" the purpose of 
the study,. the procedures, and the materials. Then, a brief back- 
ground questionnaire and the required consent and payment forms were 
. . completed. After a set of six warm-up activities and a question and 
answer session, each participant was given a loose-leaf booklet con- 
taining the 32 activity descriptions (arranged in a different random 
orde'r for each teacher). The teacher was instructed to proceed 
through the booklet of activities, completing the first 16 activities 
in the first judgment session and the remaining activities in the 
second session. 

ERJC _ . 9 



In addition to responding co the four judgment questions on the 
reverse side of each activity description, each teacher was instructed 
to "think <aloud" as they participated in the task. These verbalizaj- * 
tions-were tape recorded and later transcribed, becoming protocols 
of the judgment task. 

» » <» 

Policy-Capturing Analysis 

The primary purpose of the policy-capturing analysis was to ac- t 
ce£s the 'degree to which judgments about instructional activities 
could be modeled hy linear mathematical representations. Simple 
linear regression equations have been shown to provide, in a number 
of cases, very good explanations of judgmental Responses (e.g., Gold- 
berg, 1968). Based on the past success of this method, linear re- 
gression equations were, computed for each of the six participants. 
The five cues were treated as independent variables on which the 
ratings given to each case were regressed. In all, 24 regression 
equations were computed using programs from the Statistical Package 
for the Social Sciences. 

Two kinds of information about the judgments emerged. For each 
participant, regression weights were studied to detexjtdne which 
factors were significant predictors of that judge 1 s rating of the 
instructional activities. Second, squared multiple correlations Were 
used to estimate the proportion of variance in each judge's ratings 
explainable in terms of the five cues present in each activity descrip- 
tion. - \ . 

' f 
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Process-Tracing Analysis * * 

r » •» " , 

Procgss-tr&cing representations are based oh records of thinking- 

in^-prpgress, usually in the f&rm of verbal protocols of £he subject's * 

' * » k * 

"thinking aloud 1 ! during a task* Protocol jnalyftris is a technique 

* that has been devised to infer f rdfe these records the information- 
.processing mechanisms underlying human problem solving behavior (Newell 

* 1977)'. Akin (1979) has summarized tye* theoretical- conceptions under- 

lying this approach: ♦ \ • , 

• • All problemf-solving behavior is assumed to consist of 

transforming' a given atate of information about the 
. problem 'into another state such th^it the second state 
is closer to containing the information that describes 
& solution to the problem than the first one. The act* 
of trans forming a problem state into another one is % 
called an operation. Given this taxonomy a 'protocol 
starts with an initial information s£ate (or the pro- , 
blem description ) followed by a sfetfuenee of many inter-/ 
mediate states before ^reaching a final (or solution ) 
state. Each state can.be obtained by the application 
of an operation (or operator ) to the previous state, 
(p. 115) s * v 

Within this 4 framework, the activity judgment task can be thought 

of as a problem-solving task culminating in four judgments* related 

• » * 

to the attractiveness, appropriateness, usefulness, and effectiveness 
of each activity. In other words, we are not only interested in the 
final judgments made by the teachers but also in the thinking process 
that leads to a judgment. • „ 

The traditional procedures used for protocol analysis have been 
most strongly influenced by the work of Newell and Simon (1972). 
These researchers have sought to produce formal representations of 
thought processes that could be simulated by computer models. They 
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have attempted, this, using the problem behavior graph (PBG) A P3G 
represents a problem-solving task as that of finding the right 
sequence of operators (or path) to transform the initial state of the 
problem into the "solution s„tate. In its complete fofrn a PBG provides 
(1) a catalogue of operations that can be applied in a task environ- 
ment, (2) the circumstances under wtiich such applications are made, 
and (3) the paths developed during the search for a solution (Akin, 

1979; Newell, 1977).\ _ , ' 

As the use of process-tracing techniques has increased in recent 
*• \ ' ' 

years, a number of criticisms of "protocol analysis have been raised 

(e.g.^Nisbett and Wilson, 1977). Four criticisms are asserted most 

• * * " ' 

%. frequently. 



1. Studies have shown that wheh subjects have been 
asked to introspect about their actions, they are 
often in etror. 

2. > The verbalization of thought processes interferes 

_th and distorts the normal course' of thinking. 
» » 
3". Since "thought processes are much faster than motor 
responses (e.g., speech), verbalization,, cannot ^ 
accurately convey the richness of the cognitive J . 

processes. , v 

4. Process-tracing analyses rely uport very 'small numbers 
of subjects in each- experiment. 

• Space does not allow an extensive analysis and response to these 
criticisms here, briefly, we can say fhat many of these concerns have 
been shown to be unfounded;' • For example, .experimental analyses of 
verbal-reporting processes indicate that verbalization of oj beets cur- 
rently in memory can be accurately conveyed without distorting noral 
thought processes. -The innacurate reports cited by some researchers • 



seem to result from asking subjects to report information that was 
never directly noticed or used, thus forcing them to infer rather 
than remember mental processes (see Ericcson and Simon, 1980 for a 
review of this research). \ 

Missing data (criticism #3) seems to be W>st problematic when 



researchers study tasks that require visual and ^otor processes that 
are difficult to put into words (e.g., in architectural design) . p 
Small sample size in these experiments is greatly offset by the 
hundreds and often thousands of observations found in the protocols 
of each subject. Conclusions are generalizations about the nature of 
the cognitive task and about consistencies within the problem solver 
rather than generalizations to the population of problem solvers. 

Results 

Policy-capturing and process-tracing analyses of judgment are 
most powerful as intra- individual methods. Since both Methods 
generate a sizable amount of individual data that becomes less 
meaningful when summarized across subjects, we have chosen to present 
results for only two of six teachers who* participated in the study. 
Data from two subjects will adequately illustrate the types of infor- 
mation generated by the two methods and will be a feasable amount of 
data to attempt to present and discuss within these space limitations. 
The two teachers whose judgment processes are described in this report 
were selected because they have the same number of years of experience 
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teaching upper-elementary grades and because the proportions of 
variance in their judgments explained by their policy equations are 
representative of those of 'the six participating teachers. 

The Policy-Capturing Analysis 

Teacher 3, a female, instructs a fourth-grade class in a rural 
school; «§he has four years of teaching experience. The' mean rating 
given by this judge to all activity descriptions was 5.26 with a 
standard deviation of 2.80. The average squared multiple correlation 
from the four regression Wd-eTs-Ten-erared-for--the-four-^udgTnenfes~was 
.28 (adjusted R 2 = .14), which indicates that, as a conservative 
estimate, a little less than one-sixth of the variation in Teacher 3's 
ratings can be accounted for by the five activity features manipulated 
in the study. 

The portion of this teacher's judgment accounted for by the 
regression equations sbows a fairly consistent weighting pattern across 
the four judgments. Of the five manipulated cues (Table 1), Teacher 
3's judgments of the activity descriptions were influenced most by 
the perceived amount' of student involvement in an activity (Cue 1) 
followed by demand on the teacher (Cue A) and the fit of the activity 
with the stated purpose (Cue 5) . This teacher gave the other two cues 
no significant weighting for any judgment. 

Teacher 5, a male,' instructs a fifth-grade class in a suburban 
school; he also has four years of teaching experience. The meen 
rating given by this judge to all activity descriptions was 6. 57-\^ 
(standard deviation = 2.53). The average squared multiple correlation 
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Table 1 



Cues 



1. Involvement 

2. -Difficulty 

3. Integration 

4 . Demand 

5. Fit 



Regression Weights for Four Judgments 




Teachers 3 and 5 




Judgment 




Attractive Appropriate Likely to 


Effective 


TIs'e 





T3 


T5 


T3 


T5 


T3 


T5 


T3 


T5 


.37** 


.41 


.19* 


0 


.72* 


.19 


.40** 


.19 


- .19 


. .16 




.25 


.34 


.50 


.28 


.19 


.06 


.22 


.06 


0 


.22 


-.37 


.16 


-.08 


.12 


-.53 


.06 


-.13 




-.87 


;34*— 


— ,-1-3— 


.19 


-.41 


.19* 


-.43 


.47 


--.25 


.34* 


-.37 



*£ < .10 
**£ < .05 
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from the four regression models, was .08 (adjusted R =0). This 
indicates that the regression models of this teacher's judgment 
captured virtually none of the variation- in -judgment. This is 
further confirmed in Table 1, vhich shows that no significant regres- 
sion equations were generated for this teacher. 

The results for these two teachers, though somewhat disturbing, 
were similar to results from the other four teachers. Of the 24 
regression models generated for all six teachers, only 11 equations 

had significant cue weightings, and a majority of the regression 

/ 

r% _ , Q . , , — - 

equations had low R 's (mean R = .16; range from .03 to .35). These 
results are also consistent with another study of 19 teachers j ud ging 
these same materials (Yinger, Clark, and Mondol, Note 6) in which 44 
percent of the regression equations showed no significant regression 
weights and where the squared multiple correlations were also low 
(mean R = .18; range from .03 to .50). 

In the previously cited study, we hypothesized about the reasons 
for the poor showing of the policy-capturing models for this task. 
We suspect that a, major reason is the limitation of the model in 
trying to represent a complex naturalistic judgment in terms of only, 
a few cues. The process-tracing analysis that follows will address 
this hypothesis in more detail. 

Process Tracing ° 

The protocol analysis produced three types of information 
about the judgment activity: (1) a description of the heuristic 

16 
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operators used by the judges, (2) a reconceptualization of the task 
itself, and (3) a description of the ~ues used by the judges during 
the task. Each of these wiH be discussed in -turn. - 

Heuristic operators . Processing information during problem- 
solving tasks has been described in, a variety of ways, all of which 
emphasize the organized, deliberate, and purposeful orientation that 
people adopt in most cases, Newell and Simon (1972) characterize 
these processes in terms of operators. Others have used terms such 
as plans (Miller, Galanter, and Pribram, 1960), schemas (Akin, 1979), 
and scripts (Abelson, 1976). Often the terms reflect the level of 

analysis adopted by the res earchers. Nfiwell an d Simon reduce the 

process to b^sic operators that might be simulated on a computer, 
while thos6 lising schemas and scripts are referring to broader 
organizations of complex, temporal action. 

We have chosen to use the term heuristic operator to describe the 
processes used in this study* The word operator is used to refer to, 
the manipulations or, operations that the judge (teacher) is called 
upon to do. We preface this by the word heuristic to distinguish 
our use from the very specific operators in the Newell and Simon tradi- 
tion. Also we see these operators as being employed ir* the heuristic 
sense of framing a strategy, or approach that is used in a flexible and 
variable way. 

The initial protocol analyses indicated that the teachers were 
doing distinguishably different things at different points in the task. 
Some of these activities were repeated regularly, others occurred less 

c 

frequently. On the basis of this analysis/ approximately a dozen 



14 

"information processing routines" were initially identified in the two 
teachers' protocols; By more careful-examination ;-~of"the~ "operators 
and by subsuming some under others, ""Che Tist was reduced to seven 
heuristic operators that were widely used by both teachers. They are 
reading, interpreting, mental trying-out, categorizing, editing, evalu- 
ating, and justifying. 

1. Reading . Reading is the process of gathering information 
from the text of the activity description. Reading was the only 
source of information available to the judges and thus was a 



necessary part of the task. The participants read in a variety of 

\ 

Jw§YS^_ljicli^i!^^ 



pick up general information (e.g., the length of the description), 
and "skip reading 51 (skipping over certain information like lists or 
examples) .»* 

2. Interpreting . Interpreting occurred often and in close 
proximity to reading. It 'showed that participants made an effort to 
understand what was read. Interpreting included paraphrasing pre- 
vious reading and constructive activity — information mentioned that, 
while consistent with the text, was not required or suggested by it. 
A portion of a protocol showing the alternations between reading and 
interpretation at the beginning of an activity judgment is listed in 
Figure 1. 

3. Mental trying-out . Mental trying-out refers to visualizing 
an activity in progress while preparing to make a judgment. Basically, 
it involves mentally playing out or running through an activity 
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Interpreting 



, Heuristic Operator Line 

e 

- -Reading — 1 # 

2. 
3. 
A. 
5. 
6. 
7. 
8. 
9. 
10. 

11. 
12. 
13, 

: 14^- 

15 .„ 
16. 
17. 
18. 
19. 
20. ( 
21. 
22. 
23, 
24. 
h 25. 
2d. 
27. 
28. 
29. 
30. 
31. 
32. 
33. 
34. 
" 35. 
36. 
37. 



Reading 

Interpreting 
Reading 

Interpreting 



Commentary 



The first activity is number 28 and 
the name of the activity is Hot 
Off the Press. The purpose: After 
completing this activity, the 
student sho'uld- .be able to summarize 
in writing the current .events of 
his classroom life and present in 
several styles of writi ig the 
concepts he is learning. 
Ok, so he's going to have to sum- 
marize in writing and he's going 
to be dealing with different 
styles of- writing. Ok, so I will 
-tr-y-and— reraember~t-hosev-~ ~v 
Alright. The first .thing he'll 
do is' to. . . 

Introd.uc_e_-.the, idea_of_wxi£iiig--a_ 



newspaper to your class by bring- 
ing lots of newspapers to schopl 
for' examination by the students. 
Ok, so this would be one style' of 
writing. 

Consider together all the. parts 
and purposes of a newspaper, the 
style of writing, methods of 
reporting, etc. 
-.Ok, urn..., I don't know if the 
purpose of the activity is to 
worry about the style of writing... 
whether you-would need to go into 
the parts and the purposes of the 
newspaper — but that may be 
beneficial — or th^ methods of • 
reporting. I guess it would 
depend on how much you wanted to 
stick to the purpose of the ac- 
tivity. 



Figure 1. Reading and interpreting* in a protocol. 
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16, 

to gather information about its workability. It often involves 

picturing the activity-as- i£-~f its-^ infco-a-w±der e&ivtext-; 

De Groot (1965) first used this term in his analysis of chess 
thinking. This term was also used'by Yinger (1979, 1980) to 
characterize how teachers investigate instructional plans. Twq 
distinct types of trying-out were evident in these protocols. The 
first, projection, involved imagining the activity within the context 
o! the teacher's current modus operandi in the classroom with 
his/her students. An example from one of the protocols illustrates 
this operator: 1 

(Reading) Using three of this week 1 s spelling words, , 
tell a story about the^ magic potion to the class. 
Write sentences using these-spelling words with > - 

correct spelling and punctuation. (Projection) My 
spelling program is a little dif ferent— it 1 s not-J:hat 
the whole class has spelling words. They have their 
own spelling words. So I guess that would have to' be 
worked out. Good We f d come up with varied sen- 

tences — which may be very good.*.. 

Visualization refers tvo a more abstract or general trying out 
where the judge is not "mapping 11 the activity onto a specific 
situation but rather trying to visualize the general case. For 
example: 

I'm trying to think if fifth graders would think that 
Was too juvenile or not.... I think (pause) . .ok. 

Visualization was used less often than projection. It may be 
that trying-out is harder ta do. and JLess fruitful in the abstract. 

A. Categorizing . Categorizing, as the name suggests, involves 
the judges matching the activity description with a familiar 
activity category or type. The teachers, reduced all or portions of 
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diet Activity to categories- and types of instruction that they were 
familiar with, relying on pre^stored knowledge and opinions to come 
to understand judgment. Teachers categorized generally and specifi- 
cally. They categorized generally when they tried to' match the written 
description with a type of instructional process or activity like role 
playing, a "practice activity, 11 a "filler activity," or an "application 
activity. 11 Specific categorization involved a more precise matching 
of the activity with one the teacher had used before. The following 
protocol segment exemplifies this operation: 

OK, I've seen jthis done before--this~kind -of activityr 

And I've even done this kind of activity with my own 
students on different holidays, 
» * < 

5. Editing > Editing is ; process of mentally changing in the 

activity description during the reading and judgment pro'cess. -Editing 
most often took place in conjunction with mental trying-out or 
categorizing. Judges seemed to be strongly disposed toward, modifying 
activities into what they considered more workable forms — a goal that* 
was not a part of the task presented 'to them (more will be said on 
this point later), 

6. Evaluating . The teachers commonly made evaluatiye comments 
^N^throughout the protocols. This seemed to help the teachers prepare 

fofc^the final judgment ratings by assigning positive of negative 
weightih^s" to the activity as it was being interpreted and analyzed. 
In certain places, global, nonspecific evaluations were made, for 
instance, "I likeXhis!", "This is nice.", or "There's something wrong 
here." At other times^xthey made specific or pointedly critical 
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comments: "I -like -the idea of proofreaders" or "I don't like the 
idea of the students being independent.' 1 ( 

7, Justifying. Justifying involves presenting a rationale for 
a decision or evaluative remark. It often accompanied evaluating or 
editing, most commonly following the responses to the four judgment 
questions at the end of each activity, Mpst likely, some justifying 
was a response to the task constraints (i.e., thinking aloud^ as part, „ 
of a study of teacher judgment). But justifying also seemed to help 
the teacher explain an opinion to him/herself. By peeing how much 
support could be garnered for a criticism, modification, or evaluation 
the judges seemed to be checking their, own reasoning processes — a kind 
of self-testing. A portion -of a protocol that illustrates justifying 
in relation to evaluating and editing is shown in Figure 

To begin to understand when and how these heuristic operators 
were used by the judges, we • explored more closely how the teachers 
approached the activity- rating task. \This' is the topic of the fol- 
lowing section. 

Understanding the judgment task . 

Newell and Simon 1 s theory emphasizes that rational 
human problem solving is characterized by *being 
adapted to the problem to be solved. Since it is 
adaptive, the behavior of a* person solving a problem 
tells us more about the structure of the task than about 
* the personality dynamics of the problem solver. 
. (Shulman and Elstein, 1975, p. 14} 

Process tracing reveals important information about both the 
task and the thinking processes of those engaged in it. Earlier 
we said we were interested not only in the judgment process itself 
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Heuristic Operator 

Reading 

Interpreting 

Reading 

Interpreting 

Rereading (purpose) 

Evaluating 

< >• 

Justifying 
Editing 



Line 



1. 

2*' 

3. 

A. 

5. 

6. 

7... 

8. 

9. 
10. 
U. 
12. 
13 
1A. 
15. 
16. 
17. 
18. 
19. 
20. 
21. 
22, 
23. 
-2A.. 



Commentary 

sOk, number three. Let students,, _ 
_e*change-thei-r written' VdrlTwith one 
other student. _ 
Ok, they've written down what 
they're doing. 

Have them read and correct the paper 

- Jo r "pwnc tua tdon-arid-s pe 1-ling-. — - — - 
Ok, I guess we're getting a little 
English in there too. The purpose is 
to identify several creative uses of 

c common objects. ^ ^ 
....Several creative uses ...and 
should gain increased.^ facility in 
the processes required for critical 
thinking and. . .mature judgments. 
Ok, I wouldn't... I don't know if I 
like concentrating so much on the 
written work. 

* Because if you're trying to get 
critical thinking out of it, the * 
written work* I don't 7 think should 

"be emphasized. 
If I used this I wouldn't emphasize 
ttv,t . 



Figure 2, Evaluating, justifying, and editing in a protocol, 
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but also in how subjects prepared for judgment. In the next section 

we discuss our conceptions of the activity judgment task based upon 

* 

the protocol analysis and related research. 

Johnson (1955) characterized judgment as a final. component in a 
thought process preceded by preparation and production processes. 0 
These prior considerations becon^e necessary because judgment is based 
on some type of matching of the objects to be judged" with other 
similiar objects or representations of them (Johnson,. 1972) . Based^ 
on these notions we expected to observe the teachers engaged. in some 
kind of matching behavior as a major part of the task. * 

We also expected to observe the teachers employing processes to 
help them understand the written descriptions. Recent problem-solving 
research has indicated that understanding a problem is an important 
first step to solving a problem (e.g., Hayes a.,*d Simon, 1974; Hayes, 
tfaterman,~<and Robinson, 1977) ♦ Understanding ><as btssn defined in 
this reaearch as the process of constructing representations of a 
problem or task. (This process is an example of Newell and Simon f s 
notion of constructing a "problem space 11 within which a solution will 
be sought.) Representation of a problem based on written information 
relies upon general knowledge (syntactic and semantic knowledge) but 
also relies upon knowledge of specific problem type*. 

Initially, we conceived of the activity-judgment task as a two- 
step process: 

. 1. understand/ represent the activity description; and 
2. .make a judgment. 
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It could also be viewed in terms* of the subjects goals: judgment 
(major goal) and representation (sub-goal) • 

As the analysis progressed we realized that the ''understand, 
then judge 11 model was too simplistic. First evaluations were made 
very early for many activities and judgments were being constructed 
hand-in-hand With the construction of the activity representation* 
Also, w£ observed a sizable amount of mental, manipulation of the 
activities as the teachers attempted to construct activity repre- 
sentations and* fit them to specific activity routines or types 

(categorizing) or to compare them against their general conceptions 

* ✓ 

about learning, instruction,^ and principles of practice (mental 
* 

„ trying-out) . 

. Our current conception of the activity- judgment task describes 
it iit terms of the task goals (thecals the teachers set for them- 
selves) and the heuristic operators used to attain the goals. We 
have divided the task goals into four components: 

Understand/represent the written description (reading, 
interpreting, categorizing). 

2. Answer the questions, "What would this activity look like ' 
in practice (mental trying out)? 11 "How well would it work 
(evaluating)?" and "How well do I like it (evaluating) ? t! 

r 

3. Answer the question, "How could I make this activity work 
1 (editing, justifying)?" 

4. Make final judgments. 

This model of the task structure is not intended to be a strict 
representation of the flow of processing in the task but rather a 
breakdown of the various operations that seem to occur in the minds 



of the teachers. Specifically, Components 2 and 3 help to .explain 
much of the exploration and analysis behavior that was apparent 
prior to judgment. We think that the fout judgment questions (see • ' 
materials section) created several problems that needed to be solved 
before an., activity could be rated. The most expedient way to solve - 

the problems* was through investigating the activities using visual- 

* ... * 

ization or projection or through simplifying the new activity to a 
familiar form (categorization). In fact, there was a distinguishable 
portion of the protocol for every activity judgment that showed 
evidence of one or more of these operators. x - 

The third component in the model accounts for the editing 
behavior that was common td both teachers. The question "How could 
I make this work? 11 was' not a part* of the task, but a self-imposed 
problem-solving step. In fact / Question 3 (see materiajs section) 
•>for each activity asked the teachers to judge the likelihood that 
they would use the activity in their classroom "as it is. 11 "How could 
I make this work?" is possibly an example of the "collector s dis- ■ 
position," the tendency for professionals to collect and store problem 
representations and solutions without a currently intended use. 
(See Yinger, Note A, for a discussion of this behavior in one 
teacher's planning.) * The teachers . frequently used editing processes 

in their deliberations— in 63% of the activities judged by Te&cher 
5 and in 32% of those. of Teacher 3. 

The fourth goal, making the final judgments, was a necessary 
and expected part of the task. Previously we- said that we had 
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observed evaluating ?nd justifying being used earlier in the process . 
\ . ■ ' ' ' 0 

• ; This reinforces our notion that considerable overlap or intertwining 

of the processes existed in the accomplishment of the task goals, 

even though it makes sense to talk about them in four parts • 

* Our representations (Figure 3) of the sequence and nesting of 

» 4 

heuristic operators for Teachers 3 and 5 in their analysis of the 
/same activity are analogous to problem behavior % graphs (sea the 
section, Policy Tracing Analysis) but are not at a more general 
level. 

» *. * • 
Focus of attention in the task ,. We have thus far described the 
. judgment task as approached by che teachers and the heuristic opera- 
t'ors they appeared, to use in the process. Now we will describe 
T^ 1 aspects of the activities or their context that were present in the 

* teachers 1 thoughts during- the activity-judgtnent task. This analyses y 

/ 

is the process-tracing counterpart to the information about /due 
usage in the. policy- capturing analysis and may help to ^Illuminate the 
results obtained there. 7 

In their judgments of the activity descriptions, both teachers 
mentioned quite a few considerations or cues. Teacher 3 mentioned 
22 different cues and Teacher 5 mentioned 28. Four of the five cues 
manipulated in the activity descriptions were among those mentioned 
by each teacher; neither mentioned "integration. 11 When considering 
a single activity, each used a moderate number of cues (mean cue 
use per activity: Teacher 3, 6.81; Teacher 5, A. 75)^ Teachers 3 
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Heuristic Operators Used By 
Teacher 5 



Reading 
Scanning 

Categorizing 
Reading 

Skip-reading 
Evaluating (global) 
Evaluating (specific) 
Editing 
Projecting 
Justifying 
Evaluating (global) 
Categorizing k 
Judgment //I 

Justifying 
Judgment //2 

Evaluating (specific) 
Judgment //3 

Jus tifying 
Judgment /M 
Justifying 



Heuristic Operators Used By 
Teacher 3 



Reading 

Visualizing 
Categorizing ' 
Projecting, 
Reading 

Evaluating (global) 

Visualizing \ 
Evaluating (global) 
Reading 

Rereading 
interpreting 
Projecting 
••Interpreting 
Reading 

Evaluating (global) 
Reading 

Evaluating (global) 
Projecting 
Judgment //I 

Justifying 
Judgment //2 

Justifying 
Judgment #3 

Justifying 
Judgment #4 



Figure 3. Sequence of heuristic operators employed two 

teachers while examining and judging Activity No. 8 
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and 5 mentioned factors in their deliberations on the activities 

that could not always be considered cues of the activity de*- v 

scriptions alone* They considered many aspects of the activities 

that take on meaning only in the context of implementation. ( Since 

i ■ 

mental trying-out places activities in the context of implementation, 
we observed teachers talking about many specific and often idiosyn- 
cratic concerns refated to themselves, their students, and their 
classroom. All of the considerations for both teachers, however, can 
be grouped under 1 one of three foci: activity, teacher, or student. 

Twelve considerations or cues were used by both teachers. In 
relation to the activity itself, both mentioned fit of the activity 
process with its stated purpose, clarity of the activity description, 
activity type, and age level appropriateness of the activity. The 
cues related to students were (1) student involvement, (2) student 
enjoyment, (3) difficulty for students, and (4) effective outcomes. 
Concerns mentioned under teacher focus were (1) demand on the teacher, 
(2) fit with teacher's goals, (3) fit with current practice ("what I 
do 15 ), and (A) fit with past practice ("what we've done"), A look at 
the complete cue lists for the two teachers (Figure 5). shows that each 
considered many aspects beyond those mentioned in the activity 
descriptions as a means of accomplishing their task goals. 

Process-tracing analysis does not produce a metric comparable to 
the cue weightings derived from a regression model. We can, however,^ 
get a rough indicator of the importance of various considerations 
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Teacher 3 



Focus 



Student 



Student involvement- 
Student difficulty 
Students' task-related 

ability (2) 
Incidental learning 
Student interest (6) 
Cognitive ^outcomes 
Affective outcomes 
Student enjoyment (7) 
General stiident outcomes 
Individual differences 



Teacher 



Demand 

Fit with Teacher f s goals (5) 
Prerequisite instruction (1) 
Fit with" current practice 
"Feel" 

Fit with past practice- 



Activity 



Fit of purpose and description (3) 
Clarity of procedures 
Appropriateness, of instructional 

strategy 
Activity type 
Internal consistency 
(Age level appropriateness (4) 



Teacher 5 



Focus 



Student 



Teacher 



Activity 



Student involvement (5) 
Student difficulty ■ 
Student enjoyment (1) 
Affective outcomes 
Peer interaction 
Student choice 
Success 
Student need 
Challenge 



Demand 

Fit with current practice 
Fit with past practice 
Fit with teacher* s goals (3) 
Enthusiasm 



Fit-of purpose and description (2) 

Brevity 

Variety 

Academic defensibility 

Terminology 

Clarity 

Uniqueness (A) 

Sequence 

Design/flow 

Diagnostic opportunity 

Activity type 

Practicality 

Expansion potential 

Age-level appopriateness (6) 



Figure 5, Cue lists for Teachers 3 and 5. 
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from the number of times cues were used. (It is important to 
remember that frequency of use provides no clue as to the importance 
of a cue in a specific deliberation or how- it was used in relation to 
the other cues considered at the same time.) For Teacher 3, seven 
cues accounted for approximately 50% of the incidences of cue use 
(presented in rank order, from most frequently mentioned): (1) pre- 
requisite instruction, (2) students 1 task-related ability, (3) fit 
of stated purpose with description, (A) age-level appropriateness, 
(5) fit with teacher's goals, (6) student interest, and (7) student 
enjoyment. Cues falling into this same category for Teacher 5 (also 
rank ordered) are (1) student enjoyment, (2) fit of stated purpose 
with description, (3) fit with teacher's goals, (A) activity unique-, 
ness, (5) student involvement, and (6) age-level appropriateness'. 

It is worth noting, as we cautioned above, that frequency of use 
is not directly comparable to cue weighting. For instance, Teacher 5 
frequently used the cues of fit and involvement but these cues were 
never significantly weighted in the regression models for his 
judgments. Further comparisons of the information generated by the 
two modeling methods will be taken up in the discussion section. 

Discussion . 

' We now discuss what, we have learned in this study about (1) teach- 
er judgment of instructional activities, (2) drawing on professional 
experience, and (3) methods of modeling professional judgment.- 
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Teacher Judgment of Instructional Activities 

This study produced information about how six teachers (two of 
whom are reported on here) approached the activity- judgment task in 
terms of the problems they confronted,^ the problem-solving methods 
they used, and the. factors tfyey took into consideration. We modeled 
the teachers 1 approaches to /the task as a four-component process. It 
began with attempting to understand the activity description by 
constructing a mental representation* This representation was then 
analyzed and judged as the teachers attempted to answer questions 
about the activity's Workability and potential. Efforts were also 
made to edit and revise the activity into "betfer 11 forms*. 

This approach to the task raises the question of why the teachers 
engaged in such extensive examination and even editing of the activity 
descriptions* We propose two answers here. First, it seems most ef- 
ficient to examine a plan for action tinder simulated conditions* A 
related strategy is to see how well an activity description compares 
to plans that have been successful in the past* Both of these approaches 
have been observed in other studies of teacher planning (e.g., Yinger, 
1980; Clark and Yinger, Note 7). • 

Second, through investigation and editing, the two teachers, 
reported hete seemed to be complicating the process to simplify the 
product in order to transform the activity into either something they 
could use or something they could forget. . As a result of this, by 
the end of their participation, they may have picked up a few workable 
ideas rather, than a number of activities that would, still require a 
lot of work to implement. 
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This study also added to our knowledge about the processes 
used in teacher judgment. Seven heuristic operators were proposed: 
reading, interpreting, mental trying-out, categorizing, editing, 
evaluating, and justifying. We discussed the use of these proces- 
sing methods as a means of accomplishing the goals in the task. 

A third product of the study was a characterization of the types 
of factors considered during the judgment task. The two teachers 
mentioned in this paper considered information related to three 
foci: the activity, the students, and the teacher. Most signifi- 
cantly, the cues mentioned by the teachers in the protocols Showed 
that they were considering a broad range of factors, many of which 
were not directly related to the activity itself. In other words, 
the teachers were going beyond the information given and con- . 
sidering idiosyncratic personal and contextual information. This 
makes sense given the assumed "set toward implementation" that 
teachers would bring to examining a collection of activities of 
this sort, but it poses problems for researchers who would attempt 
to understand and model judgment in the abstract (see methodological 
discussion below). ' 

Drawing on Professional Experience 

Traditionally, most psychological research has been done on 
naive or inexperienced, subjects (e.g., college sophomores). This 
has also been true of much of the problem-solving, research that has 
used tasks £hat require little, if any, specialized knowledge or 
skill. An early exception to this was De Groot's (1965) work on 
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chess thinking, which has inspired in recent years an interest in 
expert- problem solving and how experience is brought to bear in 

professional tasks. This study revealed that teachers drew upon 

« 

their professional knowledge and experience to help them examine 
and judge activity plans. 

The protocol analysis., revealed two methods that teachers 
incorporated into .their activity-analyses* In the first, mental 
trying-out, teachers either visualized portions of or whole ac- 
tivities operating in generalized situations or projected the 
activity to their current operating context with the students in 
the classroom. Categorizing was the second operator and involved 
matching parts or all of the activity description with known types 
or categories of instructional processes or activities. 

Both mental trying-out and categorizing show that these 

i 

teachers drew heavily on their professional knowledge and experience 
in this task. This is consistent with results from other studies of 
expert problem solving in which methods were used similiar to those 
of this study. Mental trying-out (projection and/or visualization) 
is similar to the problem-investigation methods observed in^chess 
thinking (De Groot, 1965; Chase and Simon, 1973). The same pro- 
cesses have been observed in teacher planning (Yinger, 1980; Clark 

and Yinger, Note 7). Simon (1980). has suggested that methods with 

; f 

a strong perceptual component are important skills, that professionals 
develop . 
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Categorizing, described in this study, is similar to methods 
found in other studies that draw upon experience as highly organized, 
large-scale functional units. These experience units—scripts, ( 
schemas, production systems, or routine.s— have been evident in studies 
of experienced individuals solving physics problems (Simon and -Simon, 
1978; Larkin, Note 8), solving algebra word problems (Hinsley, Hayes, 
and Simon, Note 9), understanding stories (Shank and Abelson, 1977), 
and planning instructional activities (Yinger, 1979); Studies 
comparing expert problem solving with novice or beginner problem 
solving have found that novices approach problems piecemeal and are 

unable to effectively draw upon large scale organizations of know- 

/ 

ledge and experience. In terms of this study, we could expect that 
less experienced teachers .categorize,' visualize, and project more 
simply than more experienced teachers do. We would also predict 
less editing among novices. 

Methods of Modeling Professional Judgment 

The, results of this study support our earlier assertions (Yinger, 
Clark, and Mondol, Note 6) that policy-capturing methods only inade- 
quately represent complex judgment tasks. These methods have been 
shown to be much more effective in highly controlled tasks that limit 

.to a. small number of unambiguously present cues the amount of infor- 
mation available. We feel that the trade-off that these studies make 
in reducing ecological validity to gain experimental control is un- 
acceptable for studying professional judgment. The large number of 
cues revealed in the process tracing illustrates the limitations of 

' trying to understand or model the process using only fivs cues. 

')cr 
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If policy capturing seems like a limited j*ay to determine the 
important factors considered by a teacher, it says nothing at all 
about a teacher's conception of the task and the process and 
methods used to accomplish it. Policy-capturing models assume that 
judgment proceeds in a fairly straightforward manner by the additive 
combination of weighted cues. Though these models have proven to be 
highly predictive in certain judgment situations, we doubt that they 
have much explanatory value in terms of actual task conceptions or 
processes. 

Process-tracing methods are imperfect. Much of the knowledge 
that guides our day-to-day or professional activity is tacit and not 
easily verbalized. Thus, some of the information relevant to 
describing thought processes remains uncap tured. Still, the 
naturalistic, even ethnographic orientation of the method seems to 
be well suited to understanding teacher thinking. We are assuming, 
of course, that the improvement of professional practice is based 
on understanding and x not merely on prediction and that examining 
the tasks, the methods,\nd the meanings created by teachers is an 
important avenue to arrivink at this goal. 
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